Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Temporary blacklisting of queries that ends up scanning large number of rows #791

Open
wants to merge 1 commit into
base: next
Choose a base branch
from
Open

Temporary blacklisting of queries that ends up scanning large number of rows #791

wants to merge 1 commit into from

Conversation

r-santhosh
Copy link

We use opentsdb as our metric system for application and system health monitoring.

Some times, we end up having a scenario in which one of region server becomes hot spotted due to a query that scans large number of rows. This is due to the high cardinality for the metric and time range for which the query is issued.

To mitigate this scenario, we've implemented a temporary blacklisting on such queries. The code change is attached as a patch file with this email.

This change has been done in the query layer because it has the right contextual information like rows scanned to best take the decision about blacklisting. This code change will reactively blacklist queries that scan more than 'n' rows for a specific duration of time 't'. This feature can be turned off/on via config and it's off by default. Settings like max number of records ('n'), blacklist interval ('t') is configurable via config file.

Currently this change works only when the query is made using the metric name and does not handle the scenario where the query is based on TSUID.

… end up scanning large number of rowscan be blocked for some time.
@johann8384 johann8384 modified the milestones: v2.3.0, v2.4.0 Jul 6, 2016
@johann8384 johann8384 requested a review from manolama May 10, 2018 06:58
@BrokenWingsIcarus
Copy link

ok?

@johann8384 johann8384 modified the milestones: v2.5.0, 2.6.0 Oct 26, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants